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We report the findings of an empirical analysis of the performance of a group of middle 
and high school students before and after an after-school algebra enrichment program 
using the SimCalc software incorporating classroom networks. The results highlight 
statistically significant gains in their learning and briefly outline contributing factors of 
the innovation that gave rise to such improvement. 

CONTEXT 

The long-term goal of the SimCalc project has been to democratize access to the 
Mathematics of Change and Variation (MCV) (Kaput, 1994) especially algebraic ideas 
underlying calculus (Kaput & Roschelle, 1998) using a combination of new 
representations, links to simulations and new curriculum materials for grades 6-13. The 
software enables students to interact with animated objects whose motion is controlled by 
visually editable piece-wise or algebraically defined position and velocity functions. One 
form of the software has been developed for the TI-83+ (Calculator MathWorlds) and the 
other is a cross-platform Java application (Java MathWorlds) which exploits higher 
screen resolution, with the ability to pass MathWorlds documents between the two 
platforms - see http://www.simcalc.umassd.edu for further details. 

Present work has studied the integration of various kinds of connectivity with the 
SimCalc environment that enable new and intense forms of social interaction and 
learning possibilities. Recent studies (Kaput & Hegedus, 2002) reported the affordances 
of newly emerging connectivity technology that allows teachers to distribute and collect 
students’ work across diverse platforms from hand-held devices, such as the TI-83+ 
graphing calculator, to desktop computers. Since this earlier work, we have expanded the 
design space of classroom connectivity to include the passing and sharing of students’ 
individual constructions between computers using standard Internet protocols. The 
development of a dedicated Java MathWorlds (JMW) server has allowed students to send 
their individual work from within JMW to the teacher who can aggregate and then 
display their work on a single coordinate system. 

Utilizing this new connectivity/aggregation ingredient, we developed a 5-week after- 
school algebra enrichment program for local middle and high school students in a 
standard school computer lab. This paper examines their mathematical performance 
gains. 

THEORETICAL BACKGROUND 

The capabilities of classroom connectivity has not merely enhanced the management of 
information flow in the classroom, but more powerfully, changed the nature of MCV 
learning activities. We have begun to document activity structures (Hegedus & Kaput, 
2002) that give rise to mathematically deep and socially intense learning where students’ 



3-47 



personal constructions become part of shared mathematical objects as their work visibly 
participates in those aggregated objects. The social structure of the classroom plays a 
direct role in the structuring of mathematical activities, and vice-versa in a dialectical 
fashion. Students, organized into groups, build functions that vary parametrically across 
the groups as well as within groups, yielding structured families of functions reflecting 
the directly experienced social structure of the classroom. This epistemologically elevates 
the organizational structure of the mathematical objects, from functions to families of 
functions. In doing this, students construct parts of a mathematical whole and so the focus 
of their attention is on the relations between their individual contribution and the whole. 
Thus, students’ personal identities are intimately involved in their building and sharing of 
mathematical objects in the public space of the classroom. The main aim of this paper is 
to show that classroom connectivity not only offers new and exciting pedagogical 
opportunities for teachers but it can significantly improve students’ performance in core 
algebra topics over a short period of time. 

THE SET-UP OF OUR CONNECTED SIMCAUC CUASSROOM 

During the course of our intervention, the class 
met after school in a dedicated computer lab. 
Figure 1 illustrates schematically the classroom 
set-up with a large amount of activity occurring 
around the whiteboard (Display) where the 
teacher computer was displayed. Two rows of 
computers (shaded) were used for other school 
purposes except for one computer (S) dedicated 
to running the SimCalc Server application. 
Four rows of computers (16 computers) were 
used, usually with 2 students to a computer (C). 
The classroom set-up was traditional in its 
layout with rows of computers, making 
interaction and classroom discussion logistically difficult in contrast to networks of 
personal hand-helds. The focus of our attention here is on the students rather than the 
teacher - a novice SimCalc teacher who received regular direction from us. 

Our key innovation was to incorporate a unique identifier for each student into the new 
connected classroom activities. Each student was assigned a 4-digit number, which 
resonated the physical group set-up of the classroom. The first two digits specified their 
group number (established by which row of computers they were in for example) and the 
last two digits specified their count-off number in their group. This 4-digit number served 
as a natural variant, which mapped to the parametric variation within the mathematical 
activities. For example, a now standard introductory task in our work requires each 
student to construct a motion by visually or algebraically editing position-time graph 
segments to make their screen object travel at 3 feet per second for 5 seconds but starting 
at their group number. When each student’s work is aggregated, the teacher can display a 
naturally emerging family of functions (a set of parallel functions - see Figure 2) and 
begin to discuss the variation across the graphs in terms of the variation across the 
groups. 
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Figure 2. Class Aggregation in Java MathWorlds 

Overlap occurs in the position-time graphs since each group contains several students, 
but each student is distinguished by a unique square dot in the upper half of the screen 
(“the World”). Clicking on a dot displays the student’s name in the lower left part of the 
window (this can be deselected to provide anonymity). Animating the group motion leads 
to all the dots moving synchronously in the world, but offset in side-by-side groups as in 
a parade. Based on projection of the teacher’s computer onto a classroom display, class 
discussion centers on who is where and why, and why the motion and graph 
configurations appear as they do. Students are also asked to produce a generic formula 
for the entire group, for this example, Y=3X+B where B is the group number. An 
additional feature of the SimCalc software is a Matrix window, which displays all work 
retrieved from the students and allows the teacher to hide/show students’ graphs, dots and 
other representational features of the motion. In addition, and more importantly, the 
teacher can sort dots by group or count-off number, thus displaying groups of dots which 
naturally correspond to groups of students. Here, the students’ identities are projected 
into an organized aggregate structure, which resonates with the structure of the 
mathematics. In addition, the dynamic feature of the SimCalc environment enables a 
motion-representation of the generic formula and the role of the parameters M and B - 
via the parallel motion and offset starting positions. The parallelism of the linear 
functions is represented in the parade -like motion of the group! 

In later activities, we systematically vary the role of both group and count-off number in 
increasingly more rich and complex tasks dealing with core algebra topics, such as slope 
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as rate, linearity, parametric variation and systems of equations. A more detailed account 
of our intervention triangulating our empirical work with our observational data can be 
found in Hegedus & Kaput (in review). Following our intervention fusing the SimCalc 
environment and classroom connectivity with these and many more related activities, we 
saw significant gains in student performance. We now report our testing methodology 
and empirical results. 

PRE-POST TEST METHODOLOGY 

In order to measure changes in students’ understanding of the core algebra ideas attended 
to in our intervention, we administered a 20-item pre-post test comprised of 12 questions 
from the 10 th grade 2001 state (Massachusetts) exams, 1 Advanced Placement Calculus 
item and 7 questions selected from a pool of items developed by the SimCalc Project and 
refined over several years. Fourteen of the questions were multiple choice and the 
remaining items were short answer or open-response. Some of the questions were not 
directly addressed in our intervention and served as face- validity items that assessed 
whether more general algebraic skills were developed during the intervention. Twenty- 
five students from our original group of 38 completed the course. The scores of 24 of 
these students were used in our statistical analysis, as one student did not complete the 
pre-test. Those students not completing the course were not statistically different 
(mean=0.430, variance=0.019, n=13) in their achievement on the pre-test from the final 
sample used for pre-post test comparison (mean=0.427, variance=0.019, n=24). 

We adopted the rubric for the MCAS open-response item (4 points) and combined it with 
the other multiple-choice questions and SimCalc items (scored between 2 and 4 points) to 
total a test score of 31 points. The test items were comparable in difficulty and passed a 
test for internal consistency reliability (oc=0.71) for its use as a pre -post-test measure. The 
pre and post-tests were scored by two markers. A high interrater coefficient (r = 0.80) 
was obtained and a third marker was used in collaboration with the other two markers to 
obtain the final results. 

DATA RESULTS AND ANALYSIS 

We aimed to show that any change in student’s performance was mainly caused by our 
intervention of the connected SimCalc classroom and not necessarily the social or 
academic demographics of the participants or other indirect variables. To this aim, we 
present our results, which separate out the performances of 2 different subgroups and the 
performance on individual items combining various statistical procedures and measures 
of gain. Our middle school students were higher achievers (as described in their mean 
scores). Our high school students were low achievers, with low proficiency levels 
(average 218) on recent 8 th grade state examinations. The five-week teaching experiment 
had a positive effect on the mathematical behavior of both groups of students as 
presented in Table 1. 

We used a paired Student’s t-Test to measure the significance of the difference in the 
groups mean scores. A paired test was suitable given a high correlation coefficient 
between their pre/post-test scores (0.78) as well as the two groups being identical and the 
same test being used to measure the effect of the intervention. 
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Group 


n 


Pre Test 


Post Test 


Cohen’s 
d effect 


Hake’s 

Gain 


P 






Mean 


SD 


Mean 


SD 


All 


24 


42.7% 


0.141 


65.9% 


0.149 


1.60 


0.42 


0.0001 


7 th & 8 th 


10 


52.2% 


0.092 


76.8% 


0.123 


1.78 


0.5 


0.0001 


9 th 


14 


37.7% 


0.158 


62.0% 


0.136 


1.91 


0.36 


0.0001 



Table 1. Pre- and Post-Test Results 



In addition, a parametric test was used given that the students were selected from a large 
population and had varying mathematical abilities. This was confirmed using a normality 
test 1 . The results show a statistically significant increase (p<0.05) in both the mean scores 
as a group, as well as by age group. Even though the group of grade 7 & 8 students 
(n=10) showed higher test averages than the grade 9 students (n=14) the latter group 
yielded a higher effect size of 1.91 standard deviations. In both groups, the effect size 
was extremely high. This illustrates that while the 5 -week session had a very positive 
effect on both groups it appears that there was a more positive effect on the 9 th grade 
students. Our concern was with the difference in both age groups both in background and 
prior knowledge. How much of this gain was due to prior knowledge? To attend to this 
question, we used Hake’s gain statistic - an average normalized gain - which related 
mean gain relative to original performance, i.e. Gain = <Post> - <Pre> / 1 - <Pre> where 
the angled brackets represent mean scores. Hake studied over 6000 diagnostic tests of 
physics undergraduates in reform- vs. traditional-based classrooms (Hake, 1998) 
observing higher gain scores (>0.4) for reform-based classrooms. Hake’s work and other 
studies indicate that this statistic is related to students’ growth in a more cognitive sense 
(McGowen & Davis, 2001; Hake, 1998). Table 1 highlights how, while our group of 9 th 
graders had a greater effect size, our group of 7 th and 8 th graders had a greater gain (0.5) 
relative to their performance on the pre-test. We calculated an individual Hake’s gain 
statistic for each student for two purposes. First, we calculated how their increase in 
performance from pre to post test related to their prior knowledge by correlating their 
individual Hake’s gain statistic with their pre-test scores. There was insignificant 
correlation for both the middle school students (r=0.09) and our high school group 
(r=0.12) highlighting that instruction was at the right level for students who have an 
average or little prior knowledge of the subject as judged by the pre-test score. This was 
an important find for us in establishing that gain in our non-standard classroom was 
mainly based on our intervention rather than student background given that we had a 
mixture of students of varying educational performance, of varying exposure to the core 
mathematical ideas we were attending to, as well as ages. 



1 To test whether the samples were from a Gaussian distribution we used the method of Kolmogorov and 
Smirnov. 



3-51 





Test 

Item 


Type 


Pre-Test 


Post-Test 


P 


Mea 


SD 


Mea 


SD 


1 


Direct linear extrapolation. Interpreting 
straightness qualitatively. Unit conversion 


0.333 


0.482 


0.667 


0.482 


0.0195 


2 


Modeling. Reading graphs with zero starting 


0.917 


0.282 


0.875 


0.338 


>0.999 


3 


Algebraic modeling of linear quantities with 
non-zero starting values 


0.833 


0.380 


1 


0 


n/a 


4 


Concept of averaging. Unit conversion. End 
point analysis 


0.333 


0.482 


0.417 


0.504 


0.547 


5 


Graphical interpretation. Comparing starting & 
ending point differences. 


0.458 


0.509 


0.917 


0.282 


0.001 


6 


Co-linearity. Interpretation as slope 


0.333 


0.482 


0.458 


0.509 


0.375 


7 


Linear extrapolation. Multiplicative reasoning 


0.792 


0.415 


0.833 


0.381 


0.813 


8 


Attention to labels and scales. Unit conversion. 


0.667 


0.482 


0.875 


0.338 


0.109 


9 


Reading graphs 


0.750 


0.442 


0.875 


0.338 


0.313 


10 


Interpreting Pi as a coefficient 


0.333 


0.482 


0.875 


0.338 


0.0002 


11 


Point-wise interpretation of graph to obtain 
two varying quantities. Graphical construction 


0.583 


0.504 


0.625 


0.495 


0.820 


12 


Slope as rate. Interpretation of velocity (SC; 
OR4pt) 


1.917 


0.583 


2.958 


1.367 


0.002 


13 


Slope as rate, Change of sign (slope) (AP) 


0.542 


0.509 


0.583 


0.504 


0.813 


14 


Concept of rate. Multiplicative reasoning (SC, 
SA2pt) 


0.750 


0.676 


1.792 


0.588 


<0.001 


15 


Slope as rate. Average. (SC, SA2pt) 


0.125 


0.338 


0.208 


0.509 


0.491 


16 


Slope as rate attending to scales (SC) 


0.333 


0.482 


0.833 


0.381 


0.0005 


17 


Concept of rate with divisors. Non-standard. 
(SC) 


0.417 


0.504 


0.625 


0.495 


0.206 


18 


Algebraic expression of real-life varying 
quantities. Interpreting the y-intercept 


0.750 


0.847 


2.042 


1.233 


<0.001 


19 


Determining slope from value. Concept of 
families of functions (parametric variation) 
(SC, OR4pt) 


0.625 


0.824 


1.167 


0.637 


0.0012 


20 


Algebraic and geometric reasoning. 
Interpretation of real-life scenarios. Systems of 
equations . (MC AS ,OR4pt s) 


1.458 


1.141 


1.792 


1.103 


0.148 



Table 2. Item-by-item analysis 



We also conducted an item-by-item analysis to assess which questions and corresponding 
content and skills contributed to the increase in performance for the group. Table 2 
highlights the mean scores from pre- to post-test as well as content areas for each 
question, where “SA” denotes short answer and “OR”, open-response questions. The 
remainder are multiple-choice. All questions are from the 2001 State examination unless 
stated; SimCalc (SC), Advanced Placement (AP). We conducted a Wilcoxon matched- 
pairs test for multiple-choice items (binary response) and paired Student’s t-test for short 
answer and open response questions. We highlight 8 items of statistically significant gain 
(p<0.05). One ceiling-effect item (3) could not be tested. 
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The highlighted items in Table 2 outline a genre of skills, which were mainly consonant 
with our original aims for the course. They involve interpretation of graphs, 
understanding of slope as rate, interpreting Y=MX+B and how varying the parameters M 
and B vary graphical views, linearity, interpreting slope in real-life situations (e.g., 
(constant) velocity as slope of linear position-time graphs), and, most significantly, to 
generate and interpret families of linear functions from parametrically varying M and B. 
Curiously, we had not originally intended to concentrate on graphical interpretation, 
especially for non-motion based scenarios, and we believed our intervention had not 
concentrated on such a skill, yet the overall performance of the group on such tasks 
(items 1, 5, 10, 12) led us to believe that there was something implicit in our instruction 
and the connected classroom set-up that enabled students to continually reflect on the 
graphs they created (both algebraically and visually). We cannot include the whole test 
but a full version is available on-line at the SimCalc website 2 , however we wish to 
present two of these questions which serve as face-validity items to strengthen our claim 
that a connected SimCalc classroom has a positive effect on students’ learning. The 
students scored significantly higher on the post-test for item 5. Their pre-test performance 
was very similar to the mean scores for the high school (45.0%) the previous year as well 
as the State overall mean score (46.0%). This item required students to interpret and 
compare non-linear graphs of varying enrollment figures of three clubs over time. Here 
students must determine end-point differences and interpret the results as quantities. 

5. Based on the graph, which organization showed the 

most growth in membership over the 10-year period? 

A. The Math Club 

B. The Hiking Club 

C. The Drama Club 

D. The Drama Club and the Hiking Club are tied for the 

most growth. 



Figure 3. Item 5 

The second face-validity item we wish to highlight is item 10, which shows the most 
significant increase given that on the pre-test the students performed the same as the high 
school (34%) and the state (33%), which was quite poor, and excelled on the post test 
(87.5%). Furthermore, the question required the students to interpret a non-standard 
algebraic relationship of two geometric quantities. Our intervention had predominantly 
used standard algebraic notation (i.e. Y=MX+B) or some incorporation of identifiers (e.g. 
count-off/group numbers) into the parameters M and/or B. 



Club Membership, 1985-1995 




10. The circumference, C, of a circle is found by using the formula C = it d, where d is the diameter. 
Which graph best shows the relationship between the diameter of a circle and its circumference? 



2 http://www.simcalc.umassd.edu/NewWebsite/pretest.html 
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CONCLUSION 

It is our primary claim that in combining the dynamic SimCalc environment with 
classroom connectivity we can significantly improve students’ performance on 10 th grade 
MCAS algebra-related questions in a short period of time. Even though we had a non- 
standard mixture of students, our analysis has shown that all our students performed 
better relative to their prior knowledge, which in some cases was little or none, on 
questions involving core algebraic ideas. We believe that classrooms which integrate 
dynamic software environments with connectivity technology can dramatically enhance 
students’ engagement with core mathematics beyond what we thought possible in the 
absence of such support. Further work is needed, both to explain such enhancement and 
to exploit it. 
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